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Foreword 


This  paper  was  prepared  for  the  Software  Engineering  Directorate,  Research,  Development,  and 
Engineering  Center,  Army  Aviation  and  Missile  Command. 

In  accordance  to  the  Carnegie  Mellon®  Software  Engineering  Institute  (SEI)  work  plan,  this  doc¬ 
ument  is  to  be  considered  a  preliminary  description  of  a  proposed  approach  for  center  of  excel¬ 
lence  certification  and  is  submitted  to  the  Software  Engineering  Directorate,  Research,  Develop¬ 
ment,  and  Engineering  Center,  Army  Aviation  and  Missile  Command  only  for  its  use  in  further 
developing  center  of  excellence  certification  criteria. 

The  document  presents  a  proposed  approach  and  does  not  represent  a  total  SEI  endorsement  of  the 
approach. 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Abstract 


Centers  of  Excellence  (COEs)  are  created  throughout  the  federal  government  in  all  domains  to 
signify  expertise  important  for  elevating  the  significance  of  the  product  or  service  that  is  provided. 
But  how  are  these  COEs  designated,  accredited,  or  certified?  Are  there  means  of  auditing,  assess¬ 
ing,  or  appraising  them?  How  do  they  achieve  their  “Center  of  Excellence”  appellation?  The  an¬ 
swer  is  not  clear-cut  or  without  mystery.  It  is  often  a  matter  of  trust  to  accept  that  the  excellence 
declared  in  any  COE  domain  exists. 

Criteria  and  standards  to  certify  an  organization  as  a  COE  are  presented  in  this  Carnegie  Mellon 
Software  Engineering  Institute  preliminary  report.  These  standards  are  derived,  in  part,  from  the 
balanced  scorecard  business  approach  and  reflect  criteria  suggested  by  the  U.S.  Army  Aviation 
and  Missile  Research,  Development,  and  Engineering  Center,  Software  Engineering  Directorate 
(SED). 

An  assessment  approach  was  developed  employing  established  criteria  and  standards  to  certify  an 
organization.  The  assessment  approach  took  advantage  of  the  established  appraisal,  assessment, 
and  audit  methodology.  Other  assessment  approaches,  such  as  ISO  audit  techniques,  were  investi¬ 
gated  and  incorporated  as  appropriate. 
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1  Introduction 


This  paper  discusses  the  approach  and  results  in  developing  a  set  of  generalized  criteria  and  an 
evaluation  methodology  to  be  used  in  certifying  organizations  under  the  jurisdiction  of  the  Army 
Materiel  Command  (AMC)  Chief  Information  Officer  (CIO)  as  centers  of  excellence  (COE)  for 
their  respective  domains. 

The  AMC  ClO/Chief  Technology  Officer  (CTO)  expressed  a  desire  to  establish  centers  of  excel¬ 
lence  for  selected  organizations  under  his  jurisdiction. 

In  supporting  this  need,  Aviation  and  Missile  Research,  Development,  and  Engineering  Center 
(AMRDEC)  Software  Engineering  Directorate  (SED)  proposed  to  establish  a  center  of  excellence 
for  the  domain  MIMOSA.'  As  part  of  the  establishment,  AMRDEC  SED  recommended  that  each 
such  organization  be  required  to  be  “certified”  (within  their  selected  domains)  in  order  to  be  clas¬ 
sified  as  a  COE. 

The  certification  process  would  require  generalized  criteria  and  an  evaluation  methodology  using 
these  criteria.  The  concept  of  generalized  criteria  is  that  they  could  be  applied  to  candidate  COEs 
in  different  domains. 

Certification  would  be  granted  by  the  AMC  CIO/CTO  using  evaluation  scoring  data  derived  from 
third  party  (i.e.,  objective)  evaluators. 

Tasks  involved  in  developing  the  certification  process  for  centers  of  excellence  included  the  fol¬ 
lowing: 

•  researching  relevant  information  on  the  definition,  establishment  and  evaluation  of  centers  of 
excellence 

•  generating  a  framework  that  sets  the  context  for  the  criteria 

•  defining  elements  of  this  framework,  its  evaluation  methodology,  and  evaluation  scoring  cri¬ 
teria,  including 

developing  a  definition  of  a  center  of  excellence  and  other  definitions  to  support  a  con¬ 
sistent  approach  to  certifying  an  organization  as  a  COE 

developing  criteria  to  be  used  in  an  evaluation,  leading  to  certification  of  an  organization 
as  a  COE  in  accomplishing  its  assigned  mission 

developing  an  evaluation  approach  or  methodology  employing  the  established  criteria  to 
certify  an  organization  as  a  COE 

developing  a  presentation  reporting  the  results  of  the  evaluation 


1  The  Machinery  Information  Management  Open  Systems  Aiiiance  (MIMOSA)  is  a  not-for-profit  trade  association 
dedicated  to  developing  and  encouraging  the  adoption  of  open  information  standards  for  operations  and  main¬ 
tenance  in  manufacturing,  fleet,  and  facility  environments.  MIMOSA'S  open  standards  enable  collaborative  as¬ 
set  life-cycle  management  in  both  commercial  and  military  applications. 
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2  Approach 


2.1  Definitions 

In  developing  the  certification  framework  described  here,  definitions  were  required  to  set  context. 
These  definitions  are  the  structural  members  of  the  center  of  excellence  framework  and  help  to 
integrate  the  various  framework  dimensions,  associated  criteria,  and  evaluation  methodology.  The 
definitions  were  developed  to  ensure  the  criteria  and  subsequent  evaluation  methodology  could  be 
applied  consistently.  For  example,  words  such  as  quality  and  performance  are  often  used,  but  are 
seldom  defined  or  understood  by  all  in  the  same  way. 

2.2  Center  of  Excellence  Framework 

The  framework  and  the  approach  to  developing  the  appropriate  criteria  were  derived  from  ele¬ 
ments  of  both  the  balanced  scorecard  (BSC)  [Kaplan  1998]  and  the  Baldrige  Criteria  for  Perfor¬ 
mance  Excellence  [Baldrige  2008].  Both  of  these  frameworks  address  measuring  the  health  and 
performance  of  an  organization.  There  are  common  dimensions  in  both  of  these  frameworks.  Spe¬ 
cifically,  the  balanced  scorecard  methodology  is  based  on  four  perspectives  (or  dimensions)  of  an 
organization’s  performance — customer  focused,  financial,  internal  business  process,  and  learning 
and  growth.  These  are  depicted  in  Figure  1 . 


Figure  1:  Balanced  Scorecard  Dimensions 
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The  Baldrige  Criteria  for  Performance  Excellence  framework  embodies  seven  categories:  leader¬ 
ship;  strategic  planning;  customer  and  market  focus;  measurement,  analysis,  and  knowledge; 
workforce  focus;  process  management;  and  results.  We  used  the  leadership  category  for  the  COE 
framework.  In  addition,  scoring  guidelines  are  provided  against  which  an  organization’s  perfor¬ 
mance  can  be  judged  and  rated.  The  Baldrige  scoring  guidelines  contain  five  levels  of  perfor¬ 
mance  excellence  for  each  dimension  being  evaluated. 

The  framework  is  shown  in  Figure  2. 


Figure  2:  Certification  Framework 


2.2.1  Framework  Key  Performance  Areas  (Characteristics) 

The  key  performance  areas  (or  characteristics)  for  each  dimension  further  define  the  basic  defini¬ 
tion  of  a  center  of  excellence  and  provide  descriptions  of  what  the  organization  should  possess  or 
exhibit  for  that  dimension.  These  characteristics  help  set  the  context  for  the  criteria.  For  example, 
a  COE  must  have  the  capacity  and  capability  to  accomplish  its  mission.  To  make  the  criteria  more 
applicable  to  the  organizations  that  might  be  certified,  the  characteristics  were  not  focused  on  a 
specific  domain — MIMOSA,  for  example.  Rather,  these  characteristics  are  described  in  terms  of 
assigned  mission  area,  thereby  being  applicable  to  any  domain.  Also  note  that  the  characteristics 


2  Note  that  these  categories  overlap  in  concepts  to  some  extent  with  the  concepts  embodied  by  the  BSC  dimen¬ 
sions  shown  in  Figure  1 .  A  more  detailed  mapping  of  the  exact  overlap  is  beyond  the  scope  of  this  document. 
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to  be  included  are  a  mix  of  behaviors,  values,  assets,  and  other  attributes  relevant  to  that  dimen¬ 
sion.  This  implies  that  the  evaluators  for  a  COE  need  to  be  conversant  in  the  wide  variety  of  crite¬ 
ria. 

2.2.2  Framework  Criteria 

The  criteria  for  each  framework  dimension  follow  from  the  characteristics,  and  are  the  standard 
against  which  the  COE  candidate  organization  will  be  judged  as  achieving  the  level  of  excellence 
required  by  that  organization  in  each  dimension  of  the  framework.  First,  the  criteria  are  written  to 
ensure  they  are  measurable  and  observable  (the  criteria  must  include  the  phrase  “documented  evi¬ 
dence”).  This  inclusion  allows  the  use  of  an  evaluation  methodology  to  have  artifacts  rather  than 
obtaining  evidence  solely  from  interviews. 

2.2.3  Framework  Scoring  Criteria 

The  proposed  evaluation  method  follows  the  high-level  flow  of  a  process  assessment  with  the 
scoring  criteria  noted  below.  It  is  meant  to  have  an  objective,  qualified  third  party  follow  the  high- 
level  steps  of  planning,  conducting,  and  reporting  the  evaluation  results.  Actual  certification  lies 
with  the  AMC  CIO/CTO  using  evaluation  results. 

The  scoring  criteria  are  the  way  the  evaluators  rate  the  performance  excellence  of  the  organiza¬ 
tion.  Following  the  Baldrige  approach,  the  scoring  criteria  contain  five  levels  for  each  dimension 
being  evaluated.  The  characteristic  of  the  lowest  level  is  that  none  or  few  of  the  criteria  for  that 
dimension  have  been  satisfied;  at  the  highest  level,  most  to  all  of  the  criteria  for  that  dimension 
have  been  satisfied  and  that  there  is  a  measure  of  excellence  above  the  basic  criteria  that  is  ob¬ 
servably  present. 
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3  Generalized  COE  Criteria  and  Evaluation  Methodology 


3.1  Center  of  Excellence  Definition 

Research  provided  several  definitions  of  a  center  of  excellence,  a  few  of  which  were  relevant  to 
the  goals  of  AMRDEC  SED  and  AMC.  However,  most  self-proclaimed  COE  organizations  based 
their  definition  more  on  what  the  particular  center  does  (functions)  in  its  domain  (“center  of  excel¬ 
lence  for  cancer  research,”  for  example)  rather  than  using  a  more  general  definition  of  a  COE  as  a 
starting  point. 

Synthesizing  the  available  definitions,  the  authors  developed  a  general  definition,  augmented  by 
characteristics  for  each  framework  dimension  being  considered.  That  is,  the  dimensions  contain 
key  performance  areas  or  characteristics  of  a  COE  that  are  expected  to  be  found  in  the  organiza¬ 
tion.  The  additional  characteristics  help  further  define  the  requirements  for  a  COE,  thereby  lead¬ 
ing  to  definitions  of  the  criteria  for  each  framework  dimension  being  considered. 

The  general  definition,  derived  from  a  Training  and  Doctrine  Command  (TRADOC)  definition 
modified  to  focus  on  technology  development,  is  as  follows:^ 

A  center  of  excellence  is  a  premier  organization  providing  an  exceptional  product  or  service 
in  an  assigned  sphere  of  expertise  and  within  a  specific  field  of  technology,  business,  or  gov¬ 
ernment,  consistent  with  the  unique  requirements  and  capabilities  of  the  COE  organization. 

3.2  Center  of  Excellence  Framework 

Following  from  the  definition  above,  a  COE  is  an  organization  recognized  as  a  world  leader  in 
accomplishing  its  mission.  In  this  framework,  that  accomplishment  is  characterized  via  the  fol¬ 
lowing  dimensions: 

•  internal  business  process 

•  customer  focus 

•  leadership 

•  innovation  and  learning 

•  financial 

The  COE  framework  is  shown  in  Figure  2. 


3  TRADOC  defines  a  center  of  excellence  as  a  premier  organization  that  creates  the  highest  standards  of 

achievement  in  an  assigned  sphere  of  expertise  by  generating  synergy  through  effective  and  efficient  combina¬ 
tion  and  integration  of  functions  while  reinforcing  the  unique  requirements  and  capabilities  of  the  branches. 
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3.2.1  Key  Performance  Areas  and  Criteria  for  Dimensions 

3.2. 1.1  internai  Business  Dimension 

Key  Performance  Areas 

A  COE 

•  has  the  operational  work  system,'*  capacity,  and  capability  to  accomplish  its  assigned  mission 
in  an  outstanding  manner  where 

its  operational  work  system  is  aligned  with  organizational  needs  as  determined  by  the 
organization  and  is  repeatable,  integrated,  and  applied  consistently 
it  has  the  capacity  (environment)  to  accomplish  mission  or  tasks  (e.g.,  equipment,  facili¬ 
ties,  work,  and  support  processes  exist  and  are  in  use) 

it  has  the  capability  (leadership,  both  management  and  technical),  knowledge  and  skills, 
and  an  education  system  for  all  employees  exists  and  is  in  use 

•  consistently  equals  or  exceeds  established  effectiveness  and  efficiency  requirements  in  its 
measured  operational  performance  in  accomplishing  the  assigned  mission 

•  collects  and  uses  measures  to  make  management  decisions  in  managing  the  internal  business 

Criteria 

Documented  evidence  that  indicates 

•  the  assigned  mission  has  been  authorized  by  official  sources 

•  the  current  mission  and  function  statements  have  been  written  in  accordance  with  established 
standards,  have  been  approved,  and  are  accessible  by  stakeholders  as  well  as  program  staff 

•  processes  to  accomplish  the  assigned  mission  are  documented,  institutionalized,  and  are  be¬ 
ing  implemented  (institutionalization  is  the  ingrained  way  of  doing  business  that  an  organiza¬ 
tion  follows  routinely  as  part  of  its  corporate  culture) 

•  the  environment,  including  the  facilities,  is  sufficient  and  is  being  used  to  accomplish  the  mis¬ 
sion 

•  knowledge  and  skills  necessary  to  carry  out  the  mission  are  in  place  as  part  of  personnel 
training  and  evaluation 

•  an  operational  plan  is  in  place  and  being  implemented 

•  a  measurement  and  analysis  program  is  in  place  for  the  COE  candidate 

•  the  operational  performance  has  been  measured,  measures  analyzed,  and  results  of  the  ana¬ 
lyses  stored  and  made  accessible  to  key  leadership  personnel,  appropriate  stakeholders,  and 
program  staff 

•  results  of  analyses  have  been  and  are  being  used  to  manage  the  program 

•  results  of  analyses  have  been  and  are  being  used  to  identify  improvements  in  the  approach 


4  “Operational  work  system”  refers  to  the  combined  processes  used  to  produce  the  desired  products  and  servic¬ 
es.  The  term  “operational  work  systems”  refers  to  how  the  work  of  the  organization  is  accomplished. 
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3.2. 1.2  Customer  Focus  Dimension 


Key  Perfomance  Areas 

A  COE 

•  provides  measured,  customer-focused  performance^  addressing  customer  satisfaction 

•  elicits  customer  needs  or  requirements  proactively  and  collaboratively 

•  responds  in  a  timely  manner  to  customer  requests 

•  delivers  customer-defined,  high-quality  products  and  services 

•  anticipates  customer  issues  and  problems  compatible  with  the  COE  mission 

(Example  measures  and  indicators  include  customer  retention,  complaints,  customer  survey 
results,  product  reliability,  on-time  delivery,  customer-experienced  defect  levels,  and  service 
response  time.) 

Criteria 

Documented  evidence  that  indicates 

•  customer  relationships  are  established  and  maintained 

•  customer  relationships  result  in  customer  satisfaction  and  retention 

•  processes  and  mechanisms  for  customer  interaction  (e.g.,  customer-facing  web-based  sys¬ 
tems)  have  been  established  and  implemented  for 

proactive  elicitation  and  analyses  of  customer,  as  well  as  other  stakeholders  ’,  needs  and 
constraints 

translation  of  customer  needs  and  constraints  into  program  requirements  and  service  re¬ 
quirements  that  are  agreed  to  by  the  customer  (for  example,  schedules,  cost,  effort,  and 
product  requirements  such  as  functionality  and  product  quality  attributes) 
interacting  with  the  customer  for  delivery  of  services  (in  doing  this,  an  organization  be¬ 
gins  to  understand  the  customer ’s  situation  in  sufficient  depth  to  anticipate  future  needs, 
thereby  being  ready  with  new  ideas  when  the  customer  is  looking  for  them) 

•  customer  requirements  for  quality  attributes  (e.g.,  confdentiality  and  security)  are  elicited, 
analyzed  and  validated 

•  a  measurement  and  analysis  program  is  in  place  for  customer  interaction 

customer-focused  performance  has  been  measured,  measures  analyzed,  and  results  of 
the  analyses  stored  and  made  accessible  to  the  key  leadership  personnel,  appropriate 
stakeholders,  and  program  staff 

results  of  analyses  have  been  and  are  being  used  to  manage  the  program 
results  of  analyses  have  been  and  are  being  used  to  identify  improvements  in  the  ap¬ 
proach 

5  Customer-focused  performance  refers  to  performance  relative  to  measures  and  indicators  of  customers’  per¬ 
ceptions,  reactions,  and  behaviors  related  to  the  COE  mission,  and  to  measures  and  indicators  of  product  and 
service  characteristics  important  to  customers  and  their  customers.  Examples  include  customer  retention,  com¬ 
plaints,  customer  survey  results,  product  reliability,  on-time  delivery,  customer-experienced  defect  levels,  cus¬ 
tomer  engagement  level  on  “new”  problems,  and  service  response  time. 
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Examples  of  measures  include 

•  evidence  of  customer  retention  with  rationale 

•  number  of  complaints  received  per  product 

•  customer  survey  results 

•  product  reliability 

•  variance  of  on-time  delivery  for  projects 

•  customer-experienced  defect  levels 

•  service  response  time 

•  effective  service  to  customers 

These  example  measures  must  be  operationalized  by  the  organization  to  “fit”  its  domain  and  mis¬ 
sion.  For  example,  effective  may  be  defined  as  adequate  to  achieve  intended  purpose  or  desired 
result.  For  the  COE  the  term  effective  refers  to  how  well  a  process  or  a  measure  addresses  its  in¬ 
tended  purpose.  Determining  effectiveness  requires  (1)  the  evaluation  of  how  well  the  process  is 
aligned  with  the  organization’s  needs  and  how  well  the  process  is  deployed,  or  (2)  the  evaluation 
of  the  outcome  of  the  measure  used. 

3.2. 1.3  Leadership  Dimension 

Key  Performance  Areas 

A  COE 

•  has  proactive  leadership  to  create  and  promote  an  environment  for  empowerment,  innova¬ 
tion,  organizational  agility,  and  organizational  and  employee  learning 

•  has  senior  leaders  intrinsically  involved  with  setting  organization  performance  goals  and 
expectations,  setting  and  deploying  organizational  values,  establishing  short-  and  longer 
term  direction  focused  on  creating  and  balancing  value  for  customers,  their  customers,  and 
other  stakeholders 

•  has  senior  leaders  who  communicate  values,  vision,  directions,  key  decisions,  and  expecta¬ 
tions  through  the  leadership  system  to  all  employees 

•  has  senior  leaders  regularly  reviewing  organizational  performance  for  improvement  purpos¬ 
es  and  needed  actions 

•  has  senior  leaders  focus  on  sustaining  and  growing  the  organization  in  terms  of  both  the 
depth  and  breadth  of  its  expertise 

•  has  senior  leaders  who  focus  on  finding  and  sharing  new  insights  that  relate  to  both  the  prob¬ 
lem  space  and  solution  space  relevant  to  the  COE 

•  has  senior  leaders  that  embody  the  value  of  “learning  is  never  over”  through  their  visible, 
continual  learning  in  both  traditional  and  new  areas  of  enquiry 

•  has  senior  leaders  who  can  facilitate  appropriate  interactions  with  leading  academic  and 
practitioner  organizations  with  the  fields  relevant  to  the  COE  organization 
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Criteria 


Documented  evidence  that  indicates 

•  a  current  vision  statement  developed  by  the  organization  and  its  senior  leadership  has  been 
collaboratively  established  with  COE  stakeholders,  and  is  accessible  to  both  internal  (pro¬ 
gram  staff)  and  external  (customers  and  their  customers)  stakeholders 

•  current  mission  and  function  statements  have  been  written  in  accordance  with  established 
standards,  and  are  accessible  by  external  stakeholders  as  well  as  program  staff 

•  a  strategic  plan  is  in  place  and  is  being  implemented  that  addresses  the  organization ’s  vision 
and  strategy  for  advancement  and  sustainability 

current  performance  goals  and  associated  measures  traceable  to  the  strategic  plan  for 
the  COE  candidate  are  in  place 

activities  to  achieve  the  performance  goals  are  being  accomplished 

•  a  leadership  system  has  been  established  that  supports  the  establishment,  deployment,  and 
actual  activation  of  the  strategic  plan,  as  well  as  the  COE  organization ’s  guiding  principles 
and  values 

•  appropriate  progress  and  success  measures  have  been  established  to  measure  progress 
against  the  COE  strategic  plan;  the  measures  are  actively  used  as  part  of  organizational 
progress  determination 

•  senior  leaders  regularly  review  the  organization ’s  performance  (e.g.,  innovation,  operation¬ 
al,  financial,  and  others)  for  improvement  and  to  take  appropriate  action 

•  senior  leaders  communicate  values,  vision,  directions,  key  decisions,  and  expectations 
through  the  leadership  system  to  all  employees 

•  senior  leaders  are  actively  seeking  and  facilitating  appropriate  interactions  related  to  the 
COE  topic  areas  with  leading  thinkers  within  both  academia  and  industry 

3.2. 1.4  Innovation  and  Learning  Dimension 

Key  Performance  Areas 

A  COE 

•  has  the  demonstrated  and  proven  ability  to  incrementally  innovate  and  improve 

•  has  the  demonstrated  and  proven  ability  to  anticipate  the  direction  of  promising  solutions  to 
address  the  COE ’s  problems 

•  has  the  demonstrated  and  proven  ability  to  introduce  and  gain  traction  for  revolutionary  as 
well  as  incremental  innovations 

•  has  the  demonstrated  and  proven  ability  to  develop  the  transition  mechanisms  needed  to  turn 
innovations  into  accepted  technology  improvements 

•  has  the  demonstrated  and  proven  ability  to  create  and  leverage  the  value  network  needed  to 
support  introduction  and  deployment  of  innovations  that  are  relevant  to  the  COE ’s  customers 
and  their  customers 

•  has  an  environment  that  fosters  both  incremental  and  innovative  learning 
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•  develops,  institutionalizes  and  applies  leading-edge  technology  in  the  accomplishment  of  its 
COE  mission 

Criteria 

Documented  evidence  that  indicates 

•  technical  and  leadership  staff  are  actively  interacting  with  the  thought  leaders  in  the  domain 
of  the  COE 

•  technical  and  leadership  staff  are  actively  sharing  learning  from  both  internal  and  external 
interactions  across  the  community  of  interest  represented  by  the  COE 

•  technical  staff  understand  and  can  successfully  apply  leading-edge  technology  innovation 
and  transition  processes,  tools,  and  techniques  ( examples  include  mapping  innovations  into 
an  adopter  population,  establishing  and  analyzing  a  value  network,  and  performing  organiza¬ 
tional  readiness  analysis  for  different  innovations) 

•  a  comprehensive  and  proactive  training  program  tied  to  skills  and  knowledge  needed  to  ac¬ 
complish  the  mission  is  in  place  and  its  results  are  documented 

•  a  continuous  improvement  program  is  in  place  integrating  improvements  in  process  and  new 
technology  for  the  financial,  customer,  innovation,  and  internal  business  dimensions  of  the 
COE 

•  innovation  and  learning  performance  is  measured,  measures  analyzed,  and  results  of  the  ana¬ 
lyses  stored  and  made  accessible  to  the  key  leadership  personnel,  appropriate  stakeholders 
and  program  staff 

•  results  of  analyses  have  been  and  are  being  used  for  improvement  of  the  COE ’s  processes 
and  products 

3.2. 1.5  Financial  Dimension 

Key  Performance  Areas 

A  COE 

•  has  a  financial  management  system  that  supports  advocating  effectively  for  both  incremental 
and  revolutionary  innovations,  as  appropriate  for  its  domain 

•  has  an  established  financial  management  system  that  accommodates  the  organization ’s  needs 
to  accomplish  and  improve  its  mission  performance 

•  has  and  implements  a  financial  management  system  that  is  sufficiently  flexible  to  accommo¬ 
date  customer  needs  and  changes  in  customer  requirements 

•  has  a  financial  management  system  that  allows  appropriate  differentiation  of  non-recurring 
and  recurring  costs  related  to  proposed  innovations 

Criteria 

Documented  evidence  that  indicates 

•  a  financial  management  system  has  been  planned  and  established  at  sufficient  granularity  to 
permit  transparency  of  the  financial  decisions  to  the  appropriate  stakeholders. 
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associated  processes  of  the  financial  management  system^  have  been  established  and 
implemented 

•  financial  management  is  being  conducted  in  accordance  with  applicable  laws,  regulations, 
policies,  and  assigned  fiduciary  responsibility 

•  long-range  planning  for  fiscal  resources  is  being  accomplished  on  a  scheduled,  periodic  ba¬ 
sis 

•  financial  information  is  accurate  and  available  to  leadership  personnel,  internal  and  external 
stakeholders,  and  customers,  as  appropriate 

•  financial  performance  has  been  measured,  analyzed,  and  results  of  the  analyses  stored  and 
made  accessible  to  key  leadership  personnel,  and  stakeholders  as  appropriate 

•  Results  of  analyses  have  been  and  are  being  used  for  improvement  of  the  financial  manage¬ 
ment  system  and  processes  and  to  manage  the  COE  candidate 


6  Financial  performance  refers  to  performance  relative  to  measures  of  cost,  revenue,  and  market  position,  includ¬ 
ing  asset  utilization,  asset  growth,  and  market  share.  Examples  include  returns  on  investments,  value  added 
per  employee,  debt-to-equity  ratio,  returns  on  assets,  operating  margins,  cash-to-cash  cycle  time,  other  profita¬ 
bility  and  liquidity  measures,  and  market  gains. 
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4  Scoring  Criteria  and  Evaiuation  Methodoiogy 


4.1  Scoring  Criteria 

The  scoring  criteria  is  shown  in  Table  1. 


Table  1:  Scoring  Criteria 


Score 

Results 

Definitions 

0 

No  artifacts  or  other  evidence  as  required  by  the  criteria  are 
available  to  substantiate  implementation  or  interaction  in  this 
dimension. 

10-25 

Some  artifacts  or  other  evidence  as  required  by  the  criteria 
are  available  to  substantiate  implementation  or  interaction  in 
this  dimension. 

Performance  in  this  dimension  is  below  established 
requirements  for  the  COE. 

0  <  Some  <  25% 

25-50 

Many  artifacts  and  other  evidence  required  by  the  criteria  are 
available  to  substantiate  implementation  and  interaction  in  this 
dimension. 

Performance  in  this  dimension  is  equal  to  established 
requirements  for  the  COE. 

25%  <  Many  <  50% 

50-85 

Most  artifacts  or  other  evidence  as  required  by  the  criteria  are 
available  to  substantiate  effective  implementation  and 
interaction  in  this  dimension. 

Performance  in  this  dimension  is  equal  to  or  better  than 
established  requirements  for  the  COE. 

50%  <  Most  <  85% 

85-1 00  Most  to  all  artifacts  or  other  evidence  as  required  by  the  85%  <  Most  to  All  <  1 00% 

criteria  are  available  to  substantiate  exemplary 
Implementation  and  Interaction  In  this  dimension. 


Performance  In  this  dimension  Is  equal  to  or  better  than  world- 
class  benchmarks  In  the  domain  of  the  COE. 


The  levels  shown  range  from  “no  artifacts  or  documented  evidence”  to  “most  to  all”  artifacts  of 
documented  evidence  with  increasing  levels  of  performance. 
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4.2  Evaluation  Methodology 


The  evaluation  methodology  follows  high-level  steps  of  planning,  conducting  the  evaluation,  and 
reporting  results.  Lower  level  processes  and  activities  that  apply  to  the  evaluation  are  shown  in 
Table  2. 


Table  2:  General  Evaluation  Activities 


Phase 

Process 

Plan  and  prepare  for  evaluation 

Analyze  requirements 

Develop  evaluation  plan 

Select  and  prepare  team 

Obtain  and  inventory  initial  objective  evidence 

Conduct  evaluation 

Prepare  participants 

Examine  objective  evidence 

Document  objective  evidence 

Verify  objective  evidence 

Validate  objective  evidence 

Report  results 

Deliver  evaluation  results 

At  least  two  qualified  evaluators  must  be  used  for  the  evaluation  process.  Qualification  is  meant 
to  include  expertise  in  the  domain  of  the  COE  candidate  as  well  as  the  knowledge  of  the  criteria 
and  evaluation  methodology  described  here. 

The  rating  approach  characterizes  the  achievement  and  satisfaction  of  criteria.  Rather  than  simply 
“checking”  for  adherence  to  model  or  standard  practices,  the  COE  evaluation  method  uses  the 
criteria  as  the  evaluation  reference  model  along  with  the  scoring  criteria  shown  in  Table  1.  That 
is,  for  each  framework  dimension,  the  evaluation  looks  for  documented  evidence  of  achieving  the 
criteria  and  rates  the  achievement  by  assigning  points  according  to  the  scoring  criteria. 

For  example,  in  the  dimension  of  customer  focus,  if  most  to  all  artifacts  are  present  and  the  per¬ 
formance  can  be  shown  to  exceed  the  criteria,  including  a  judgment  by  the  evaluators  that  the 
COE  is  performing  at  the  level  of  world-class  benchmarks  in  the  COE’s  domain,  the  evaluators 
assign  85-100  points  depending  upon  the  extent  and  quality  of  the  documented  evidence.  While 
there  may  be  some  subjectivity  associated  with  the  exact  point  score  assigned,  rationale  as  to  why 
that  particular  point  value  was  given  must  be  provided. 

The  evaluators  must  include  a  characterization  of  the  quality  of  the  documented  evidence  as  part 
of  the  evaluation  where  quality  is  defined  by  the  following. 

Documented  evidence 

•  describes  measures  of  performance  used  to  substantiate  claims  for  accomplishment  of  each 
criterion  in  the  dimension 

•  provides  sufficient  details  and  substantive  information  to  convey  the  performance  achieved 
for  each  criterion  in  the  dimension 

•  exhibits  clarity  by  excluding  the  use  of  ambiguous  language  or  words  with  multiple  meanings 

•  is  internally  consistent,  with  no  conflicts  visible  to  an  external  reader  across  document  sec¬ 
tions  and  across  all  relevant  documents  and  other  evidence  in  the  dimension  being  evaluated 


16  I  CMU/SEI-2009-TN-011 


After  all  dimensions  are  scored,  a  generalized  Kiviat-type  diagram  similar  to  Figure  3  is  devel¬ 
oped: 


Customer  focus 


Innovation  & 

Internal 

Learning 

Business 

Figure  3:  Performance  Excellence  Scoring 

Based  on  the  scoring  criteria,  85  or  more  points  would  indicate  excelling  in  that  dimension.  If  all 
dimensions  are  assigned  85  or  more  points  each,  this  is  an  indication  that  the  organization  is  a 
COE  for  its  respective  domain. 

The  information  resulting  from  the  evaluation  is  provided  to  the  candidate  COE  sponsor.  The 
candidate  can  elevate  the  findings  to  the  AMC  CIO  for  certification  as  desired. 
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5  Summary 


The  authors  developed  a  framework  integrating  elements  of  the  balanced  scorecard  and  the  Bal- 
drige  Criteria  for  Performance  Excellence  to  form  a  background  for  evaluating  organizations  as 
centers  of  excellence.  The  framework  includes  definitions,  additional  characteristics  of  COEs, 
associated  criteria,  an  evaluation  method,  and  a  presentation  format  for  the  results  of  the  evalua¬ 
tion. 

The  evaluation  method  follows  high-level  steps  of  planning  the  evaluation,  conducting  the  evalua¬ 
tion,  and  reporting  results.  The  method  employs  a  set  of  scoring  or  rating  criteria  appropriate  to 
the  COE  certification  process,  and  format  for  the  presentation  of  evaluation  results. 
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